[feat](fe) Support Cloud partition inverted format rollout by hoshinojyunn · Pull Request #66235 · apache/doris

hoshinojyunn · 2026-07-29T09:58:07Z

What problem does this PR solve?

Issue Number: None

Related PR: None

Problem Summary:

Cloud tables previously used one table-level inverted_index_storage_format, so an online V2-to-V3 migration required all partitions to move together. This prevents a gradual rollout and makes it impossible to retain the correct tablet schema for old partitions while creating new partitions with a new format.

This PR adds the Cloud-only table property partition.inverted_index_storage_format. It is the format default for partitions created after the property is set. The property supports V2 and V3; V1 is rejected. Existing partitions and their tablet metas are not rewritten.

Design:

Persist the resolved base-index inverted-index storage format in PartitionInfo for every Cloud partition. The value is durable in journal/replay and is transferred through partition replacement, recycle/recover, add-partition-like, insert overwrite, and truncate flows.
New explicit, automatic, and dynamic partitions use the current partition format default. Existing partitions continue to use their recorded format, allowing V2 and V3 to coexist in one table.
A replacement partition reads the source partition's actual tablet metas and preserves each source index schema, schema hash, and schema version. This prevents truncate and insert overwrite from accidentally applying a newer table schema or format to an old partition.
A Cloud schema change collects the distinct active base-index formats. It allocates one base shadow schema version per format, persists the format-to-version map in CloudSchemaChangeJobV2, and creates each partition's base shadow tablet with the matching format and schema version. Non-base indexes keep their normal schema-change behavior.
SHOW PARTITIONS, SHOW PARTITION <id>, SHOW PARTITION ID, and the partitions TVF expose InvertedIndexStorageFormat from PartitionInfo; they no longer need to read tablet meta from MetaService for this value.

Example SQL and expected effect:

CREATE TABLE inv_format_rollout (
    k DATE NOT NULL,
    v VARCHAR(100),
    INDEX idx_v (v) USING INVERTED
)
DUPLICATE KEY(k)
PARTITION BY RANGE(k) (
    PARTITION p_old VALUES LESS THAN ("2024-01-01")
)
DISTRIBUTED BY HASH(k) BUCKETS 1
PROPERTIES (
    "replication_num" = "1",
    "inverted_index_storage_format" = "V2"
);

ALTER TABLE inv_format_rollout
SET ("partition.inverted_index_storage_format" = "V3");

ALTER TABLE inv_format_rollout
ADD PARTITION p_new VALUES [("2024-01-01"), ("2025-01-01"));

SHOW PARTITIONS FROM inv_format_rollout;

Selected output:

+---------------+----------------------------+
| PartitionName | InvertedIndexStorageFormat |
+---------------+----------------------------+
| p_old         | V2                         |
| p_new         | V3                         |
+---------------+----------------------------+

Both formats remain queryable during the rollout:

INSERT INTO inv_format_rollout VALUES
    ("2023-01-01", "old token"),
    ("2024-01-01", "new token");

SELECT CAST(k AS STRING), v
FROM inv_format_rollout
WHERE v MATCH_ANY "token"
ORDER BY k;

The query returns both rows. TRUNCATE TABLE ... PARTITION p_old and INSERT OVERWRITE ... PARTITION(p_old) continue to create V2 tablet metas for p_old, even after the default changes to V3. A subsequent MODIFY COLUMN creates distinct base shadow schema versions for active V2 and V3 partitions, then preserves the format/version pairing when committing the schema change.

Release note

Cloud OLAP tables can roll out inverted-index file storage formats by partition. New partitions can use V3 while existing V2 partitions remain online and queryable.

Check List (For Author)

Test
- Regression test
- Unit Test

Feature and Test Coverage

Function point	Expected behavior verified	Coverage
Create table and explicit partition rollout	Existing V2 partitions stay V2 after changing the default; later range and list partitions use V3 or a subsequently selected V2 default. Mixed-format `MATCH_ANY` queries return rows from every partition.	Cloud regression: `test_partition_cloud_inverted_index_storage_format_rollout`
Initial partition format override	A table created with table format V2 and `partition.inverted_index_storage_format=V3` creates its initial partition with V3.	Cloud regression: `test_partition_cloud_inverted_index_storage_format_rollout`; FE creation/property tests
Auto and dynamic partition creation	Automatically created partitions retain the format that was active at their creation time. After changing the default, newly created auto/dynamic partitions use V3 while existing partitions remain V2.	Cloud regression: `test_partition_cloud_inverted_index_storage_format_rollout`
Partition observability	`SHOW PARTITIONS`, `SHOW PARTITION <id>`, and tablet meta report V2/V3 for range, list, auto, and dynamic partitions.	Cloud regression: `test_cloud_show_inverted_index_storage_format`; FE `ShowPartitionIdCommandTest`, partitions TVF output test
`TRUNCATE TABLE` / partition replacement	Replacement creates new tablet IDs but retains the source partition's resolved format, even after the table default switches to V3.	Cloud regression: `test_partition_cloud_inverted_index_storage_format_recycle_truncate` and rollout suite; FE `OlapTableTest.testReplacePartitionPreservesInvertedIndexStorageFormat`
`INSERT OVERWRITE`	The temporary replacement partition copies the source partition's tablet schema, schema version, schema hash, and V2/V3 format instead of using the current table default. The rewritten data remains searchable.	Cloud regression: `test_partition_cloud_inverted_index_storage_format_rollout`; FE Cloud catalog and FrontendService tests
Drop, recycle, and recover	Partition and table recovery restore the persisted V2/V3 partition format and keep inverted-index queries valid.	Cloud regression: `test_partition_cloud_inverted_index_storage_format_recycle_truncate`; FE `CatalogRecycleBinTest.testRecoverPartition`
Schema change and rollup	A mixed V2/V3 table creates one base shadow schema version per active format. Each partition gets the matching version and format after `MODIFY COLUMN`; shadow base and rollup indexes are validated while the job is RUNNING.	Cloud regression: `test_partition_cloud_inverted_index_storage_format_rollout`; FE `CloudSchemaChangeJobV2Test`
Persistence and replay	Per-partition format is journaled with partition creation, replacement, and table-property changes, including legacy-format resolution and tablet-meta validation.	FE tests for property analysis, partition persistence/replay, Cloud tablet-meta creation, and schema-version validation

Executed tests:

Changed FE test classes: 70 tests passed.
Targeted OlapTableTest and CatalogRecycleBinTest: 41 tests passed.
Cloud Docker regressions:
- test_cloud_show_inverted_index_storage_format
- test_partition_cloud_inverted_index_storage_format_recycle_truncate
- test_partition_cloud_inverted_index_storage_format_rollout
Behavior changed:
- Yes. Cloud tables accept partition.inverted_index_storage_format; it affects only subsequently created partitions and exposes the resolved format in partition metadata commands.
Does this need documentation?
- No. The SQL contract and rollout behavior are documented in this PR description.

Check List (For Reviewer who merge this PR)

Confirm the release note
Confirm test cases
Confirm document
Add branch pick label

### What problem does this PR solve? Issue Number: None Related PR: apache#57013 Problem Summary: Cloud tables need to roll out inverted-index V2 and V3 formats partition by partition without changing existing partitions. Persist each logical partition's base format through creation, replay, replacement, recycle, recovery, restore, and table-property replay. Replacement partitions retain their source tablet schema. For a base-index schema change, retain the standard global target column schema while generating tablet schemas with each partition's persisted format. The partition format property is Cloud-only and is rejected in non-Cloud create and alter paths. ### Release note Cloud tables support partition-level V2/V3 inverted-index format rollout. ### Check List (For Author) - Test: Regression test and unit test - Regression test: test_partition_cloud_inverted_index_storage_format_rollout passed - Regression test: test_partition_cloud_inverted_index_storage_format_recycle_truncate passed - Regression test: test_backup_restore_inverted_idx passed - Unit test: ./run-fe-ut.sh --run org.apache.doris.common.PropertyAnalyzerTest,org.apache.doris.nereids.trees.plans.CreateTableCommandTest,org.apache.doris.service.FrontendServiceImplCloudTest,org.apache.doris.cloud.datasource.CloudInternalCatalogTest,org.apache.doris.alter.CloudSchemaChangeJobV2Test,org.apache.doris.catalog.TablePropertyTest,org.apache.doris.catalog.EnvTest passed - Unit test: ./run-fe-ut.sh --run org.apache.doris.alter.CloudSchemaChangeJobV2Test,org.apache.doris.catalog.OlapTableTest,org.apache.doris.persist.ModifyDynamicPartitionInfoTest passed - Build: DISABLE_BUILD_UI=ON ./build.sh --fe passed - Behavior changed: Yes. Cloud partitions retain their creation-time format while new partitions use the current configured format. - Does this need documentation: No

hello-stephen · 2026-07-29T09:58:13Z

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

What problem was fixed (it's best to include specific error reporting information). How it was fixed.
Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
What features were added. Why was this function added?
Which code was refactored and why was this part of the code refactored?
Which functions were optimized and what is the difference before and after the optimization?

hoshinojyunn · 2026-07-29T10:59:12Z

run buildall

hello-stephen · 2026-07-29T12:28:44Z

TPC-H: Total hot run time: 29746 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 3230373d27326ba41a7225ac142f74d7865b495f, data reload: false

------ Round 1 ----------------------------------
============================================
q1	17683	4106	4041	4041
q2	2004	331	211	211
q3	10279	1416	842	842
q4	4720	483	341	341
q5	7688	859	581	581
q6	245	172	136	136
q7	796	821	606	606
q8	10477	1651	1589	1589
q9	6227	4377	4378	4377
q10	6816	1767	1477	1477
q11	506	352	328	328
q12	766	597	455	455
q13	18111	3437	2737	2737
q14	275	264	238	238
q15	q16	793	776	713	713
q17	1024	1020	945	945
q18	6762	5947	5601	5601
q19	1387	1319	1068	1068
q20	794	679	625	625
q21	5686	2651	2537	2537
q22	441	353	298	298
Total cold run time: 103480 ms
Total hot run time: 29746 ms

----- Round 2, with runtime_filter_mode=off -----
============================================
q1	4521	4383	4403	4383
q2	282	321	214	214
q3	4623	5016	4428	4428
q4	2078	2198	1395	1395
q5	4440	4284	4276	4276
q6	234	178	129	129
q7	2403	1909	1602	1602
q8	2500	2220	2202	2202
q9	8340	7840	7671	7671
q10	4722	4681	4243	4243
q11	585	436	391	391
q12	770	810	540	540
q13	3287	3629	2923	2923
q14	290	297	274	274
q15	q16	697	729	634	634
q17	1386	1364	1492	1364
q18	8093	7285	7393	7285
q19	1177	1128	1099	1099
q20	2231	2211	1937	1937
q21	5265	4608	4390	4390
q22	531	444	423	423
Total cold run time: 58455 ms
Total hot run time: 51803 ms

hello-stephen · 2026-07-29T12:40:06Z

TPC-DS: Total hot run time: 177881 ms

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 3230373d27326ba41a7225ac142f74d7865b495f, data reload: false

query5	4339	628	496	496
query6	479	228	207	207
query7	4890	569	333	333
query8	348	190	181	181
query9	8785	4169	4128	4128
query10	480	370	324	324
query11	5951	2366	2144	2144
query12	158	104	102	102
query13	1285	602	443	443
query14	6269	5212	4889	4889
query14_1	4319	4286	4243	4243
query15	212	209	189	189
query16	1068	491	451	451
query17	1147	725	592	592
query18	2678	505	359	359
query19	219	195	157	157
query20	121	110	109	109
query21	241	165	134	134
query22	13623	13538	13399	13399
query23	17452	16481	16180	16180
query23_1	16278	16203	16289	16203
query24	7423	1795	1289	1289
query24_1	1312	1335	1308	1308
query25	571	472	415	415
query26	1338	377	220	220
query27	2523	631	388	388
query28	4421	2039	2045	2039
query29	1076	631	504	504
query30	346	265	235	235
query31	1140	1109	986	986
query32	116	64	62	62
query33	535	334	266	266
query34	1177	1161	646	646
query35	767	786	678	678
query36	1036	1038	909	909
query37	161	109	93	93
query38	1891	1678	1667	1667
query39	923	890	855	855
query39_1	835	842	867	842
query40	248	165	142	142
query41	66	62	64	62
query42	93	92	90	90
query43	326	318	279	279
query44	1469	794	766	766
query45	202	185	177	177
query46	1128	1274	768	768
query47	2119	2112	2001	2001
query48	383	426	292	292
query49	580	421	314	314
query50	1064	432	349	349
query51	10602	10618	10663	10618
query52	86	89	73	73
query53	273	275	212	212
query54	300	241	224	224
query55	75	71	65	65
query56	298	303	280	280
query57	1322	1311	1173	1173
query58	275	260	251	251
query59	1585	1654	1439	1439
query60	316	283	258	258
query61	152	149	157	149
query62	544	498	437	437
query63	244	205	198	198
query64	2799	1054	872	872
query65	4687	4646	4524	4524
query66	1815	523	386	386
query67	29454	29247	29093	29093
query68	3183	1556	968	968
query69	421	320	278	278
query70	928	828	780	780
query71	370	343	309	309
query72	3258	2741	2466	2466
query73	846	802	448	448
query74	5068	4880	4707	4707
query75	2547	2535	2157	2157
query76	2349	1191	763	763
query77	353	389	292	292
query78	12016	11891	11408	11408
query79	1422	1146	763	763
query80	803	564	504	504
query81	487	345	292	292
query82	569	162	122	122
query83	367	352	306	306
query84	275	164	131	131
query85	960	624	546	546
query86	377	250	235	235
query87	1841	1812	1771	1771
query88	3764	2842	2806	2806
query89	434	392	341	341
query90	1910	203	203	203
query91	204	190	168	168
query92	62	61	56	56
query93	1649	1566	1064	1064
query94	620	361	307	307
query95	784	591	485	485
query96	1156	816	381	381
query97	2633	2634	2477	2477
query98	217	224	215	215
query99	1095	1113	979	979
Total cold run time: 263751 ms
Total hot run time: 177881 ms

hello-stephen · 2026-07-29T12:45:07Z

ClickBench: Total hot run time: 25.08 s

machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 3230373d27326ba41a7225ac142f74d7865b495f, data reload: false

query1	0.01	0.01	0.01
query2	0.10	0.05	0.05
query3	0.26	0.14	0.13
query4	1.60	0.14	0.14
query5	0.24	0.23	0.22
query6	1.24	1.15	1.03
query7	0.03	0.00	0.00
query8	0.05	0.04	0.03
query9	0.39	0.30	0.32
query10	0.55	0.58	0.55
query11	0.19	0.14	0.13
query12	0.18	0.14	0.15
query13	0.46	0.47	0.49
query14	1.02	1.00	1.00
query15	0.60	0.60	0.59
query16	0.31	0.32	0.31
query17	1.08	1.09	1.07
query18	0.23	0.21	0.23
query19	2.14	2.02	1.92
query20	0.02	0.01	0.01
query21	15.45	0.20	0.13
query22	5.01	0.05	0.06
query23	16.16	0.31	0.12
query24	2.97	0.44	0.34
query25	0.12	0.04	0.04
query26	0.74	0.20	0.15
query27	0.04	0.04	0.04
query28	3.50	0.93	0.55
query29	12.49	4.16	3.30
query30	0.28	0.16	0.16
query31	2.77	0.60	0.32
query32	3.23	0.60	0.49
query33	3.17	3.33	3.20
query34	15.58	4.25	3.58
query35	3.50	3.53	3.50
query36	0.58	0.43	0.42
query37	0.10	0.07	0.07
query38	0.05	0.04	0.03
query39	0.04	0.03	0.03
query40	0.18	0.16	0.15
query41	0.09	0.03	0.04
query42	0.04	0.03	0.03
query43	0.05	0.04	0.04
Total cold run time: 96.84 s
Total hot run time: 25.08 s

hello-stephen · 2026-07-29T14:30:17Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.69% (31953/42217)
Line Coverage	60.33% (356043/590170)
Region Coverage	56.98% (299030/524770)
Branch Coverage	58.39% (134637/230571)

hello-stephen · 2026-07-29T14:39:50Z

FE Regression Coverage Report

Increment line coverage 49.34% (225/456) 🎉
Increment coverage report
Complete coverage report

airborne12

Two blocking lifecycle and rolling-upgrade issues are called out inline.

airborne12 · 2026-07-30T05:21:25Z

+                TInvertedIndexFileStorageFormat invertedIndexFileStorageFormat =
+                        origIdToInvertedIndexFileStorageFormat.get(origPartId);
+                if (invertedIndexFileStorageFormat != null) {
+                    idToInvertedIndexFileStorageFormat.put(newPartId, invertedIndexFileStorageFormat);


Blocking: this remaps the partition-owned format in FE metadata, but CloudRestoreJob.createReplicas() still builds every restored Cloud tablet with localTbl.getInvertedIndexFileStorageFormat() and localTbl.getBaseSchemaVersion(). After restoring a mixed V2/V3 table, PartitionInfo can therefore say that a partition is V2 while its new TabletSchema is created from the current table-level V3 format/schema. Please pass the restored partition's resolved format and corresponding backup schema/version into createTabletMetaBuilder() so the physical tablets and partition metadata are restored consistently. This needs coverage for both full-table restore and partition-only restore with mixed V2/V3 partitions.

airborne12 · 2026-07-30T05:21:25Z

        TAddOrDropPartitionsResult result;
        try {
-            result = masterCallWithRetry(client -> client.addOrDropPartitions(request),
+            result = masterCallWithRetry(client -> client.addPartitionsForInsertOverwrite(request),


Blocking: this introduces a new Thrift method without a rolling-upgrade compatibility path. During an FE rolling upgrade, a new client can call an old FE that does not implement addPartitionsForInsertOverwrite, resulting in TApplicationException.UNKNOWN_METHOD. Retrying the same method cannot recover, so INSERT OVERWRITE through a remote Doris catalog will fail until every target FE has been upgraded.

Please add explicit capability/version negotiation, or evolve the existing addOrDropPartitions request with an optional source-preserving flag and gate its use on server support. A blind fallback to the old method is not safe because it would create the temporary partition from the current table schema instead of preserving the source partition's physical schema. Please also add a new-client/old-server compatibility test.

hello-stephen · 2026-07-30T07:52:34Z

BE Regression && UT Coverage Report

Increment line coverage 100% (0/0) 🎉

Increment coverage report
Complete coverage report

Category	Coverage
Function Coverage	75.69% (31952/42217)
Line Coverage	60.32% (356013/590170)
Region Coverage	56.98% (299001/524770)
Branch Coverage	58.38% (134618/230571)

deardeng · 2026-07-30T08:33:11Z

        return partition;
    }

+    public OlapFile.TabletMetaCloudPB getTabletMeta(long tabletId) throws DdlException {


[major/P1] Avoid issuing one synchronous Meta Service RPC per partition while holding the table write lock.
collectPartitionInvertedIndexFileStorageFormats() performs O(partition_count) sequential GetTablet RPCs under olapTable.writeLock. For tables with thousands of partitions, the ALTER can hold the table metadata lock for a long time. Any RPC failure aborts the operation before journaling, and retrying starts the full scan again. Meta Service rate limiting only turns this into latency or failure and does not make the operation scalable. Please batch the tablet-meta reads and perform network I/O outside the table write lock, then reacquire the lock and validate that the table/partition metadata has not changed before committing.

hoshinojyunn requested review from 924060929, CalvinKirs, dataroaring, englefly, gavinchou, luwei16, morningman, morrySnow, mymeiyi and starocean999 as code owners July 29, 2026 09:58

hoshinojyunn changed the title ~~[feature](fe) Support Cloud partition format rollout~~ [feat](fe) Support Cloud partition format rollout Jul 29, 2026

hoshinojyunn changed the title ~~[feat](fe) Support Cloud partition format rollout~~ [feat](fe) Support Cloud partition inverted format rollout Jul 29, 2026

airborne12 reviewed Jul 30, 2026

View reviewed changes

deardeng requested changes Jul 30, 2026

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[feat](fe) Support Cloud partition inverted format rollout - #66235

[feat](fe) Support Cloud partition inverted format rollout#66235
hoshinojyunn wants to merge 1 commit into
apache:masterfrom
hoshinojyunn:feat/partition

hoshinojyunn commented Jul 29, 2026 •

edited

Loading

Uh oh!

hello-stephen commented Jul 29, 2026

Uh oh!

hoshinojyunn commented Jul 29, 2026

Uh oh!

hello-stephen commented Jul 29, 2026

Uh oh!

hello-stephen commented Jul 29, 2026

Uh oh!

hello-stephen commented Jul 29, 2026

Uh oh!

hello-stephen commented Jul 29, 2026

Uh oh!

hello-stephen commented Jul 29, 2026

Uh oh!

airborne12 left a comment

Uh oh!

airborne12 Jul 30, 2026

Uh oh!

airborne12 Jul 30, 2026

Uh oh!

hello-stephen commented Jul 30, 2026

Uh oh!

deardeng Jul 30, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

hoshinojyunn commented Jul 29, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What problem does this PR solve?

Release note

Check List (For Author)

Feature and Test Coverage

Check List (For Reviewer who merge this PR)

Uh oh!

hello-stephen commented Jul 29, 2026

Uh oh!

hoshinojyunn commented Jul 29, 2026

Uh oh!

hello-stephen commented Jul 29, 2026

Uh oh!

hello-stephen commented Jul 29, 2026

Uh oh!

hello-stephen commented Jul 29, 2026

Uh oh!

hello-stephen commented Jul 29, 2026

BE Regression && UT Coverage Report

Uh oh!

hello-stephen commented Jul 29, 2026

FE Regression Coverage Report

Uh oh!

airborne12 left a comment

Choose a reason for hiding this comment

Uh oh!

airborne12 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

airborne12 Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

hello-stephen commented Jul 30, 2026

BE Regression && UT Coverage Report

Uh oh!

deardeng Jul 30, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

hoshinojyunn commented Jul 29, 2026 •

edited

Loading